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This paper examines our use of knowledge-creation activity as a way of developing evidence- 
informed practice among a learning community of 36 early years practitioners in one London 
borough. It also seeks to illustrate how we approached the idea of measuring evidence use 
and our engagement with, and adapted use of, two separate measurement scales: the ‘ladder 
of research use’ and Hall and Hord’s (2015) Levels of Use scale. In doing so we examine the 
‘trustworthiness’ of our approaches to measuring evidence use, which we explored via in- 
depth semi-structured interviews. Our findings would appear to be encouraging, suggesting that 
knowledge-creation activity provides an effective way of communicating research and keeping 
it top of mind; also that our interview data would appear to support the trustworthiness of 
our measurement scales as a means to ascertain levels of evidence use. At the same time the 
approach we have developed does have its limitations, namely that it is only really applicable 
to situations where researchers are working regularly with practitioners on areas of practice 
development, where the general desire is that these areas should become evidence-informed. 
However, we suggest that, in school systems such as that in England - where the expectation 
is that schools or alliances of schools should lead their own professional development activity, 
often in partnership with universities - it is likely that these instances will soon be increasing 
in number. 
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Objectives 

This paper has three key aims. First, it examines our attempts to use knowledge-creation activity 
as a way of developing evidence-informed practice among a learning community of 36 early years 
practitioners in one London borough. Second, situating the idea of effective evidence use within 
Flyvbjerg’s notion of expertise, it seeks to illustrate how we approached the idea of measuring 
evidence use; specifically, how we sought to ascertain whether early years practitioners, having 
been continually engaged in knowledge-creation activity across a school year, were developing 
expertise as evidence users. As part of this second aim, we examine our engagement with and 
adapted use of two separate measurement scales: the‘ladder of research use’ and Hall and Hord’s 
(2015) Levels of Use scale. Finally, we explore how we sought to examine the ‘trustworthiness’ 
of our approaches to measuring evidence use, via the use of in-depth semi-structured interviews. 
We conclude by assessing our approach, examining both its strengths and limitations, and also 
highlighting other contexts and situations in which it might be used. 
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Perspectives 

Evidence-informed practice 

We define evidence as including the following: the use of formal research, practitioner enquiry 
such as action research, and the interpretation of routinely collected data used to inform any 
action taken. More specifically, we take the notions of formal research and practitioner enquiry 
to comprise the collection of data that has been gathered via a process of research, that has 
been interpreted, and that subsequently has or could be used to address a particular issue facing 
policy-makers or practitioners. Routinely collected data would include that already captured 
by systems within schools and school systems, such as pupil assessment data (Brown, 2013). 
Using evidence to aid practice is not without controversy or debate (e.g. see Hargreaves, 1996; 
Hillage et a/., 1998;Tooley and Darby, 1998; Biesta, 2007; Nutley et a/., 2007; Cooper et at., 2009; 
Brown, 2013). Notwithstanding these issues, this paper is grounded in the idea that engaging with 
evidence is socially beneficial: that using evidence can develop our understanding of the world 
via an exposure to ideas and concepts we might not ordinarily come across and that, in turn, this 
exposure potentially enhances the repertoire of‘understanding’ upon which we can draw as we 
develop solutions to the issues of practice. 

In itself, being evidence-informed is a somewhat nebulous notion and it is difficult to 
conceptualize what evidence-informed practice might‘look like’. It is unlikely, for example, that 
evidence will ever inform practice in a ‘pure’ form; that acts of practice will be instigated and 
will follow exactly what was intended by the researcher. Instead, evidence will conjoin with 
contextual and other practice-related factors and any decision made will be a function of all 
of these (Virtanen and Tynjala, 2008; Rexvid et a/., 2012; Marz and Kelchtermans, 2013). This is 
reflected by England’s Department for Education (DfE), which deploys the term ‘evidence-based 
practice’ (EBP) and suggests that EBP represents a combination of practitioner expertise and their 
knowledge of the best external research, and evaluation-based evidence. 

The DfE’s notion of EBP, however, highlights three key issues in relation to the concept: 

• a need to conceive of how such combinations of formal and practitioner-held knowledge 
might be meaningfully realized (i.e. from a theoretical perspective, what represents 
successful combination of knowledge types) 

• to think of approaches that will enable us to combine formal and practitioner-held 
knowledge in accordance with this theory, so that a state of being ‘evidence-informed’ 
is achieved 

• to devise ways of measuring whether we have reached our destination - that is, based 
on our theoretical perspectives and practical approaches, how might we ascertain 
whether practitioners are now informed by evidence? 

We now use the remainder of the paper to discuss these issues in more detail before examining 
our attempts to tackle them as part of the Early Years Learning Community project. 


Expertise 

We suggest that an effective way to conceptualize the combination of knowledge types described 
above is by considering notions of‘expertise’ in evidence use, a position that has its basis in the 
work of Bent Flyvbjerg. Flyvbjerg (2001) employs the ‘Dreyfus’ model of learning to illustrate 
what he means by expertise; an approach that employs five ‘levels’ of human learning, with each 
level comprising recognizably different behaviours in relation to performance at a given skill: 
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1. novice 

2. advanced beginner 

3. competent performer 

4. proficient performer 

5. expert. 

A novice,for example, is new to particular situations and will, during instruction, learn about facts 
that correspond to the situation and its characteristics, so is taught or develops ‘rules for action’. 
Flyvbjerg (2001: I I) suggests that, for the novice: 

Facts, characteristics, and rules are defined so clearly and objectively ... that they can be 
recognised without reference to the concrete situations in which they occur. On the contrary, the 
rules can be generalised to all similar situations, which the novice might conceivably confront. At 
the novice level, facts, characteristics, and rules are not dependent on context: they are context 
independent. 

Flyvbjerg argues that, as learners advance from ‘novice’ and through the levels of ‘advanced 
beginner’,‘competent performer’, and‘proficient performer’, a number of things occur to facilitate 
the normalization of more instinctual/intuitive behaviour. First, instances of performing in real-life 
situations increase so that, correspondingly, the number of ‘cases’ the learner encounters and 
tackles also increases. Second, recognition of different situations accumulates, as does recognition 
of the context in which those situations occur.Third, dependency on specific ‘rules for action’ 
diminishes as learners are able to interpret and judge how to perform optimally in any given 
situation. Genuine expertise, however, only occurs as the result of a ‘quantum leap’ in behaviour 
and perception, from that of an analytical problem-solver to someone who ‘[exhibits] thinking 
and behaviour that is rapid, intuitive, holistic, interpretive ... [expertise that] has no immediate 
similarity to the slow, analytical reasoning which characterizes rational problem-solving and the 
first three levels of the learning process’ (Flyvbjerg, 2001: 14). In other words, true expertise 
represents a situation in which experience and formal knowledge are intuitively and holistically 
combined in order that a problem might be immediately assessed and a solution provided almost 
without conscious reasoning. 

In terms of practitioner evidence use, it is clear that a spectrum of potential evidence-use 
types is likely to exist.This ranges from novice use - where practitioners slavishly follow ‘what 
works’-type evidence, mandates, or solutions produced on the back of this evidence - to expert 
use. At this advanced level, practitioners are social actors who intuitively develop responses 
to situations using an intuitive, holistic reading based on a combination of the formal research 
knowledge they have adopted to date, an understanding of the specific case they are dealing with, 
and their understanding of the other environmental factors that might influence their decision. 
We would argue that the most effective types of use stem from evidence being employed in 
this expert way. It is most likely to avoid the difficulties reported by a number of professions 
in marrying centrally prescribed, ‘evidence-based’ solutions with local context - for example, 
Rexvid et al. (2012) examine doctors’ and social workers’ reactions to initiatives to implement 
‘evidence-based’ guidance. It is also most likely to avoid any substantial costs in terms of the 
re-contextualization required with such marrying (see Cartwright, 2013, who argues that such 
costs relate to the support factors required to implement solutions in a given setting). By this 
we do not mean that the ‘best available’ evidence is slavishly followed. Rather, we suggest it is 
applied alongside practitioner understanding/data/enquiry in order to frame or define an issue, 
provide an understanding for how it might be addressed, and/or help ensure that data might be 
analysed most effectively. A fundamental difference, then, between top-down evidence-informed 
solutions (see Moss, 2013, for an excellent example) and practitioners’ expert evidence use is 
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that, in the latter, the practitioner engages with the formal knowledge that best suits their needs, 
circumstances, or priorities. In other words, formal research is filtered through a ‘contextual 
window’ which determines what research might be considered (for instance, with regards to the 
nature of the issue and acceptable ways of tackling it). 


Professional learning communities 

Having established a theory that accounts for how knowledge might be meaningfully combined, we 
now consider approaches that enable formal and practitioner-held knowledge to be combined in 
ways consistent with Flyvbjerg’s theory of expertise. Here we turn to the notion of professional 
learning communities.While there is no universal definition of a professional learning community, 
they are usually depicted as a context in which people involved with, and concerned about, 
schooling work collaboratively to learn about how they can improve pupil learning or outcomes. 
For example, such communities have been described by Stoll (2008: 107) as a means through 
which to build ‘learning [in order] to support educational improvement’, and which comprise 
‘inclusive, reflective, mutually supportive and collaborative groups of people who find ways, inside 
and outside their immediate community to investigate and learn more about their practice’.The 
notion of professional learning communities thus encapsulates instances where practitioners 
and researchers might collaborate to facilitate learning about and from formalized/academic 
knowledge. 

A key benefit of the professional learning community may be attributed to the nature of the 
learning that takes place within it. This is encapsulated by the process of knowledge ‘creation’, 
described by Stoll (2008) as one where the producers and users of formal knowledge - who are, 
respectively, also the users and holders of‘practical’ knowledge - come together to create ‘new’ 
knowledge (Stoll, however, uses the term ‘animation’ to describe the collaborative process). 
Nonaka and Takeuchi (1995) conceptualize this process of creation as one arising from the 
interactions between tacit and explicit (or informal and formal) knowledge. In particular it is 
a ‘spiralling’ that accrues from four sequential types of knowledge conversation, culminating in 
practitioners ‘internalizing’ newly created knowledge and intuitively drawing upon it as part of 
the day-to-day process of developing practice-related solutions. In other words, it results in a 
state of expertise. 

Activities to facilitate knowledge-creation activity are considered in detail in Stoll (2009), 
and we outline our approach in a general sense further below. 


How might we measure evidence use? 

A third issue relates to how we might know when any state of being ‘evidence-informed’ has 
been achieved. Researchers’ ability to measure the effectiveness of methods aimed at facilitating 
evidence-informed practice has been problematized by Levin (2013: 6-7), who argues that: 

even if one recognizes multiple forms of research influence, it is very difficult to identify when 
these have taken place or how much effect they have had ... Most of the research assessing 
research impact relies on one asking people what they did and why, even though it is known that 
self-reports, whether of belief or behaviour are not reliable. 

and that: 

Practitioners themselves cannot readily disentangle the role of particular influences on complex 
activities such as professional practice or policy choices. 
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In other words, measures of research use focus on discrete instrumental uses of research (Landry 
et o/.,2003).We argue that the issues raised by Levin are further compounded when incorporating 
the need to conceive of evidence use as something intrinsically related to expertise. This is 
because, as will be illustrated below, the levels used to describe expertise seem fiendishly difficult 
to translate into something that can be surveyed.This presents a tricky problem in relation to 
our above description of the notion of expertise and its relationship to being evidence-informed: 
How might we measure evidence use in a way that helps us ascertain whether individuals are 
learning/developing as expert evidence users? 

We now describe our approach to this issue as part of the Early Years Learning Community 
project conducted in one London borough. 


Data sources 

The Early Years Learning Community project 

In England, early years provision has undergone significant and unprecedented investment 
and intervention from governments in recent years. This is in recognition of the critical part 
that early education can play in improving the life chances of children, particularly those from 
disadvantaged groups. However, provision for children under 5 is offered within a range of highly 
diverse settings including nursery classes, private and voluntary nurseries, children’s centres and 
primary schools. Historically, the fragmented and patchy nature of this provision has created 
difficulties and divisions for children, their families, and practitioners alike. In order to overcome 
some of these divisions, the Early Years Foundation Stage (EYFS) was established in 2008 and 
revised in 2012 (DfE, 2014).The EYFS was devised with the following aims, to: 

• set the standards for children’s learning, development, and care from birth to statutory 
school age at 5 

• improve quality and consistency in the early years sector 

• lay a secure foundation for future learning through individual learning and development 
planned around the individual needs and interests of the child 

• provide equality of opportunity 

• create the framework for partnership working (DfE, 201 I). 

In England, qualifications of staff working with young children are highly variable and match 
the diversity of provision in the sector. Latest figures suggest that 73 per cent of staff working 
in childcare settings are qualified to Level 3, the equivalent of an A level in England. A much 
lower number, 7 per cent, are qualified to Level 6, or graduate level. In nursery and reception 
classes within primary schools the figure for graduate-level qualification is higher, with around 
40 per cent having qualified teacher status (DfE, 201 I). Considerable progress has been made to 
improve the qualifications of those working with young children. A further challenge, however, 
is that of ensuring continuing professional development during a period of significant change in 
the policy landscape and in curricular and pedagogical requirements. It is now widely accepted 
that good pedagogical leadership makes a difference to the quality of early years provision and 
in particular to creating an ‘enabling environment’ for learning as defined by the EYFS (see, for 
example, Siraj and Hallet, 2014). An enabling environment is one that supports children’s play and 
exploration, and their critical and creative thinking. Rogers (2013) has argued that the concept of 
an ‘enabling environment’ does not make sufficiently explicit the importance of relationships and 
interaction in teaching and learning. Hence she proposes that pedagogical leadership requires 
practitioners to think in terms of an ‘enabling pedagogy’, based on the available research evidence 
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of the interactive strategies that best meet the complex learning needs of children in the early 
years.The project reported here draws upon that idea in developing a model for professional 
development. 

Against this background, the project responded to a specific identified need to improve 
the skills and knowledge of a diverse group of practitioners in relation to the requirements of 
the EYFS. In particular the participating local authority and a number of heads of early years 
settings in the borough argued that support was required with regards to the newly introduced 
‘Characteristics of effective learning’, which provide the context for learning in the EYFS (DfE, 
2012).These are: 

• playing and exploring - children investigate and experience things, and ‘have a go’ 

• active learning - children concentrate and keep trying if they encounter difficulties 

• creating and thinking critically - children have and develop their own ideas, make links 
between ideas, and develop strategies for doing things. 

The main objectives of the community therefore were to improve outcomes for children and 
adults by: 

• increasing practitioners’ understanding of and confidence in using the newly introduced 
‘Characteristics of effective learning’ 

• enhancing early years practitioners’ teaching skills, particularly in promoting children’s 
‘Creating and thinking critically’: children’s ability to ‘have their own ideas’, ‘make 
links’, and ‘choose ways to do things’ - this focus was informed by research evidence 
indicating that this characteristic is linked to self-regulation, creativity, and motivation 
(Whitebread e£ a/., 2009) 

• developing a sustainable model of practitioner development and partnership that will 
support children’s learning, particularly those in disadvantaged groups. 

In order to realize these objectives a professional learning community was established comprising 
36 practitioners from 18 early years settings whose headteachers and setting managers were 
keen to support the opportunity for developing the skills of their practitioners in this way.The 
practitioners worked in either children’s centres catering for children from birth to 4, or primary 
school reception classes working with children aged 4 to 5. Level of qualification ranged from 
Level 2 nursery assistants to qualified teachers.The aim of the learning community was to assist 
participating practitioners (two from each setting) in developing a wider repertoire of interactive 
strategies which better support children’s creative and critical thinking. It was envisaged that such 
strategies would be formed from a combination of formal (academic) knowledge and praxis- 
based best practice, and their enactment would be observed, critiqued, and improved upon 
throughout the year via facilitated lesson study - a process involving groups of practitioners in 
collaborative planning, teaching, observing, and analysis of the learning and teaching in research 
lessons (Lesson Study UK, 2014). The efficacy of these strategies in meeting their intended 
outcomes for practitioners and children was then established via an evaluation process where 
baseline data was collected at the beginning of the project, intermediary data was collected 
at monthly intervals, and endline (outcome) data was collected at the project’s 12-month 
conclusion. At the same time the authors sought to use the evaluation process to test, first, the 
learning-community approach as a means of facilitating evidence-informed practice and, second 
(vitally), as a way of testing potential ways of measuring the success of our approach. In other 
words, to ascertain whether the knowledge-creation approach had facilitated the development 
of expertise in terms of evidence use. 
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Approach to knowledge creation 

A knowledge-creation workshop was held during the inaugural professional learning community 
meeting. Following Nonaka and Takeuchi (1995) and Stoll (2009), one of the researchers - an 
early years expert - facilitated a discussion that centred on current academic knowledge in 
relation to effective early years practice, covering the age phase from birth to starting school.This 
process had, in itself, been constructed via a detailed literature review, specifically undertaken for 
the project (see Rogers and Brown, 2014).The first session included current research findings 
on early attachment, attunement, and reciprocal play, and on the development of theory of mind, 
metacognition, and emergence of pretend play and language. A second session considered this 
research in relation to pedagogy in early years settings and schools. It included, for example, the 
latest research on sustained shared thinking, adult strategies for supporting child-led play, and 
developing an enabling pedagogy (Rogers, 2013). 

The practitioners were then invited to share their own practical knowledge, for example, 
data and insight about their settings and current practices, via activities designed to surface this 
knowledge (see Rogers and Brown, 2014).They were then invited to establish what they wanted 
to achieve by the end of the project and how they might do so. Specifically, following an approach 
set out by Earley and Porritt (2014) and starting with the end in mind (the goal they wished to 
achieve), practitioners were asked how they might relate the presented academic knowledge 
to their own practical knowledge and that provided by others in their study group, in order to 
develop strategies towards reaching this end-point. 

Following this initial workshop, practitioners from the 18 partner settings were paired into 
six groups of three from a mix of children’s centres and reception classes.This was a deliberate 
strategy, which allowed all practitioners on the project to observe pedagogy and learning across 
the entire early years age range and to strengthen networks between different early years 
provision in a sector that is widely recognized to be fragmented. Each setting hosted one visit 
per term for their study group. 

During each visit, these mixed study groups (and facilitator) spent the morning in the setting 
observing a specific activity planned by the host practitioner.These activities were designed to 
be part of normal pedagogical practice, to ensure the least disruption to the children but at the 
same time to allow the practitioners to explore new strategies stemming from the research. 
Before each activity, the host practitioner briefed observing practitioners on what they wanted 
to achieve and on desired outcomes.The group then reflected on the ‘success’ of the activity, 
offering suggestions for how it might be improved and planning the next lesson study. In order 
to ensure research evidence remained ‘top of mind’, the facilitators were asked specifically to 
ensure that, as well as improvements grounded in practitioner knowledge, they prompted for 
potential improvements suggested by the research. 

At the end of each lesson study day, all 36 participating practitioners attended an end-of- 
day seminar to ensure that learning was shared and reflected upon across the group. Where 
pertinent, the researchers would relate practitioner experiences shared during the feedback 
back session to the research initially presented. 


Methods 

The overall purpose of the methodological approach described in this paper (as distinct from 
those approaches employed as part of the overall evaluation) was to ascertain how effective 
the knowledge-creation process was in instilling an evidence-informed ethos within a specific 
group of practitioners.To do so required us to establish ways of measuring impact in ways that 
addressed earlier critiques of this approach, as noted above. Specifically, as well as addressing 
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Levin’s points, we wished to measure the extent to which practitioners had become expert in 
using evidence. As can be seen from Figure I, however (see page 254), the descriptors of each 
of Flyvbjerg’s levels of expertise seemed tricky to operationalize; it was difficult to turn these 
into scales that would be meaningful to respondents, which tackled the specific situation we 
were asking them to consider, and would enable them to employ evidence as part of developing 
strategies to improve their practice. Ultimately, therefore, this quest led us to approach the issue 
of measurement in two distinctive ways, with a subsequent phase of in-depth semi-structured 
interviews and observation work, to provide additional insight and data regarding the validity and 
reliability of our favoured approach. 


The ladder of research use 

Our first approach saw us draw upon a study successfully undertaken by Landry e£ al. (2003), 
who surveyed government officials from Canadian and provincial public administrations in order 
to examine the extent to which they employed academic research as part of the policy process. 
This study based its measurement of research use on an adapted version of the Knott and 
Wildavsky (1980) ‘ladder of research’ use (see Landry et al., 2003: 194); one recognized for 
its reliability (see, for example, Cherney et al., 2012). Correspondingly, Landry e£ al.’ s study is 
one grounded in the assumption that evidence use must be examined in terms of cumulative 
levels, or stages, of both cognitive process and action. In other words, evidence use does not 
just happen, it occurs as a process comprising many stages via a movement from its ‘reception’ 
(‘I received the university research pertinent to my work’) to ‘influence’ (‘university research 
results influenced [my] decisions’), via the stages of‘cognition’,‘discussion’,‘reference’, and ‘effort’ 
(‘I made efforts to favour the use of university research results’).This notion that use occurs 
gradually over time and is dependent on its being meaningfully‘received’ or engaged with led us 
to feel that the scale would provide a good basis from which to ascertain how well research was 
initially communicated in the knowledge-creation workshop, and the extent to which users were 
inclined to employ the research findings. 

Our scale was amended, however, to reflect the situation in which it was to be applied: 
that we were studying practitioners not policy makers, and asking practitioners to consider 
the research just presented which they had been asked to employ in the development of their 
strategies in their study groups.This led us to employing the following descriptor terms: 

1. Reception - the research was well communicated 

2. Cognition - I understood the findings of the research 

3. Discussion - I discussed with others within my study group how the research might be 
used 

4. Reference - I could relate the research findings to my setting 

5. Effort -1 used the research in subsequent exercises (when thinking about the approaches 
I might use in my setting) 

6. Influence - I intend to apply/have applied the research as part of my approach. 

In order to collect data against our scale, a self-completion survey was employed at the end of 
the knowledge-creation workshop, with the researchers in situ as the survey was completed so 
that we might address any queries regarding interpretation. Specifically, respondents were asked 
to consider all six elements and indicate the extent to which they agreed using five-point Likert 
items ranging from ‘strongly agree’ to ‘strongly disagree’.The five-point scale provided subtlety 
of understanding since it is grounded in the notion of knowledge-use being a process containing 
multiple stages, rather than an event that can be measured in a yes/no binary way. 
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Respondents were also asked to respond to additional open-response questions, so that 
we could ascertain what other factors had influenced their choice of strategies.These questions 
were: 


• ‘Why do you want to use the research?’ 

• ‘What else was being discussed by the study group?’ 

• ‘Did you also incorporate this into your strategies? If yes, how?’ 

• ‘Was there anything else that influenced your choice of strategies? If yes, please state 
what.’ 

Our initial survey thus recorded practitioners’ early thoughts about the strategies they hoped 
to create, and enabled us to ascertain the extent to which these had been influenced by the 
research that had been shared and engaged with, along with other potential influences and the 
relative strength of such influences. In total, 33 of 36 participants completed the survey. 


Levels of Use scale 

While our scale appeared to be useful for assessing the effectiveness of the knowledge-creation 
activity as a means of communicating research and as a process of stimulating interest and the 
desire to use evidence, it failed in terms of providing an ongoing measure. In particular, it did 
not enable us to determine whether practitioners’ level of expertise in using evidence increased 
over time. In other words, it did not allow us to understand the effectiveness of the evidence 
combination we had produced. We could not judge, for example, whether practitioners were 
being guided slavishly by the evidence or had combined it in a more intuitive, holistic way with 
their wider practical experience so that their overall ability to understand or tackle particular 
situations had been enhanced. It also failed to provide a way of assessing changes in expertise 
over time so that we could not judge, for example, whether practitioners had begun as novices 
and gradually improve their expertise. 

As a result, our second approach to measuring evidence use involved attempts to find and, 
if required, amend a scale that could be more closely related to Flyvbjerg’s levels of expertise. 
Following Cherney et al. (2012) we sought out scales with an ability to identify how utilization 
is related to various ongoing, complex, and indeed socially situated decision-making processes. 
Our attempts culminated with a decision to use the Levels of Use scale developed by Hall and 
Hord (2015) as part of their wider work in developing the Concerns-Based Adoption Model 
(CBAM),a tool developed in the late 1970s in order to explore people’s reaction to change and, 
in particular, to change caused by the introduction of new innovations.The Levels of Use scale 
represents eight classifications relating to how people act or behave in response to a newly 
introduced change or innovation. Specifically, it reflects the observation that, just because it has 
been suggested or mandated that an innovation should be used, it doesn’t mean that this will 
necessarily be the case, or that all individuals will employ the innovation in the same way; some 
users will ‘stumble along’ and others may use the innovation to achieve ground-breaking ends 
(Loucks et al., 1975).This idea - that there are different typologies of use occurring as a function 
of users’ ability to incorporate a new innovation into their practice - led us to believe that the 
Levels of Use scale could provide a way of understanding how notions of expertise might be 
expressed. In addition, it enabled us to develop a scale that draws on the essential features of Hall 
and Hord’s existing and extensive learning: for example, their insight that an eight-point scale is 
more likely to capture the subtlety of the different ways in which innovations are used (Loucks 
et al., 1975). 
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Figure I: Congruence between Flyvbjerg (2003) and Hall and Hord (2015) 


Flyvbjerg: Expertise that accrues through 
learning 


Non-use 


Novice 

i. ‘rigid adherence to taught rules or plans’ 

ii. no exercise of‘discretionary judgment’ 


Advanced beginner 

i. limited ‘situational perception’ 

i.all aspects of work treated separately with equal 

mportance 


Competent 

i. ‘coping with crowdedness’ (multiple activities, 
accumulation of information) 

i. some perception of actions in relation to goals 

ii. deliberate planning 
v. formulates routines 


Proficient 

i. holistic view of situation 

i. prioritizes importance of aspects 

ii. ‘perceives deviations from the normal pattern’ 

v. employs maxims for guidance, with meanings that 
adapt to the situation at hand 


Expert 

i. transcends reliance on rules, guidelines, and 
maxims 

ii. ‘intuitive grasp of situations based on deep, tacit 
understanding’ 

ii. has ‘vision of what is possible’ 

v. uses ‘analytical approaches’ in new situations or 

n case of problems 


Hord and Hall: Levels of use concerning 
a new innovation 


Non-use 

The user has little or no knowledge of the 
innovation, no involvement with the innovation, 
and is doing nothing towards becoming involved 


Orientation 

The user has recently acquired or is acquiring 
information about the innovation and/or has 
recently explored or is exploring its value 
orientation and its demands upon user and user 
system. 


Preparation: The user is preparing for first use 
of the innovation 


Mechanical use: The user focuses most 
effort on the short-term, day-to-day use of the 
innovation with little time for reflection. Changes 
in use are made more to meet user needs than 
client needs.The user is primarily engaged in a 
stepwise attempt to master the tasks required to 
use the innovation, often resulting in disjointed or 
superficial use. 


Routine: Use of the innovation is stabilized. Few 
if any changes are being made in ongoing use. Little 
preparation or thought is being given to improving 
innovation use or its consequences. 


Refinement: The user varies the use of the 
innovation to increase the impact on clients 
[pupils/students] within immediate sphere of 
influence.Variations are based on knowledge of 
both short- and long-term consequences for 
clients [pupils/students]. 


Integration: The user is combining her/his own 
efforts to use the innovation with related activities 
of colleagues to achieve a collective impact on 
clients within their common sphere of influence 


Renewal: The user re-evaluates the quality of 
use of the innovation, seeks major modifications 
of or alternatives to the present innovation to 
achieve increased impact on clients [pupils/students], 
examines new development in the field, and 
explores new goals for her/himself and the system. 


As a result, Hall and Hord’s approach and scale had an impact on our understanding as to how 
expertise might be measured. For example, as Figure I indicates, while there does seem to be 
a level of congruence between Flyvbjerg and Hall and Hord, the expertise scale as it currently 
stands has to be ‘stretched out’ since certain stages of expertise appear to apply to a number 
of Hall and Hord’s categories. Conversely, the language used in the Hall and Hord scale required 
substantial modification so that it effectively captured the expertise being displayed by given types 
of use. For example, that a user was able to modify and augment an innovation in order to obtain 
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greater impact implies a high level of connection between formal knowledge (the intervention) 
and tacit knowledge (practical understanding of the situation in hand); it implies a high level 
of expertise. Of similar interest was whether modifications were undertaken consciously or 
unconsciously, with the latter suggesting even greater levels of expertise. At the same time, 
this modification meant that we were able to tackle some of the critiques aimed at this scale, 
such as Landry et al.'s (2003) suggestion that the Levels of Use scale focuses on specific uses of 
research: we ensured that our revised scale - which drew on but did not simply replicate the 
essential features of Hall and Hord’s work - looked at changes in behaviour occurring as a result 
of practitioners’ ways of engaging with research as opposed to actual instances of research use. 
To be clear, the purpose of the work was to produce a new scale, not simply to use the existing 
scale offered by Hall and Hord which did not quite meet the needs of the study in question. Our 
new and adapted scale effectively combines two existing pieces of work to measure evidence 
use among, in this case, early years practitioners. We deploy here the parts that best enabled us 
to measure expertise in evidence use. 

Correspondingly we developed the following descriptors based on Hall and Hord’s Level 
of Use scale that sought to elicit notions of expertise in relation to evidence use (see Figure 2). 


Figure 2: Typologies and descriptors of expertise 


Hall and Hord 

Flyvbjerg expertise 

New typology and descriptors 

based on Hall and Hord and seeking to elicit expertise 

Non-use 

Non-use 

NON-USE: 1 haven’t attempted to use any of the findings from the 
research within my day-to-day practice. 

Orientation 

Non-use 

ORIENTATION: 1 have begun to consider how to use some of 
the findings of the research as part of my day-to-day practice. For 
example, in terms of specific strategies suggested by the research. 

Preparation 

Non-use to Novice 

PREPARATION: 1 have made concrete plans to use some of the 
research findings as part of my day-to-day practice and am waiting 
for the opportunity to do so. 

Mechanical use 

Novice to Advanced 
beginner 

NOVICE: 1 have now begun to implement specific strategies 
suggested by the research within my day-to-day practice. 

Routine 

Competent 

COMPETENT: 1 now regularly use some of the research findings 
within my day-to-day practice. For example, 1 frequently use specific 
strategies suggested by the research. 

Refinement 

Proficient 

PROFICIENT: 1 have begun to tailor my use of the research findings 
so that 1 can incorporate other aspects of effective practice that 1 
know about. 

Integration to 
Renewal 

Proficient to Expert 

PROFICIENT+: 1 regularly adapt the strategies suggested by the 
research in order to make them even more effective or so that they 
can apply to a number of situations within my setting. 


Expert 

EXPERT: 1 frequently use the research strategies we learnt about 
but 1 now incorporate them into my day-to-day practice in an 
automatic rather than conscious way, as 1 tackle specific situations 
and issues. 


Again, respondents were asked to consider all eight elements and to indicate the extent to which 
they agreed with them, using a scale that ranged from ‘strongly agree’ to ‘strongly disagree’.This 
enabled us to establish a more nuanced understanding as to whether a level of expertise had 
been reached and, as a result, an understanding of‘typologies’ of evidence use being displayed 
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by respondents.The survey was administered in December 2013, some four months after the 
beginning of the project. In total, 34 of 36 participants completed the survey. 


In-depth semi-structured interviews 

To explore evidence-use typologies emerging from the survey in more detail, we conducted in- 
depth semi-structured interviews.The purpose of the interviews was two-fold: first, to explore 
the actions and behaviours of individuals at different levels of expertise (as indicated via their 
survey responses): second, to explore the ‘trustworthiness’ of the scale by asking questions to 
assess comprehension of the questions and whether the questions were measuring what they 
were designed to measure. In total, 10 interviews were undertaken. 


Results 

Adapted ladder of research use 

Beginning with results from our amended ladder of use scale, it can be seen in Table I that all 
respondents agreed or strongly agreed that the research was well communicated (stage I); 
comprehension (stage 2) was also high, with 94 per cent agreeing or strongly agreeing that they 
understood what had been communicated (no respondents disagreed or strongly disagreed 
with this statement). This pattern continues until stage 4 of the ladder (‘I can see how the 
research related to my setting’), where 3 per cent disagree; correspondingly, this same 3 per 
cent disagreed that they felt they had started to use the research in subsequent exercises (stage 
5) or had any intention of using the research moving forward (stage 6). Conversely, 97 per cent 
suggested they had actively discussed the research presented (stage 3); 94 per cent could see 
its relevance and had discussed the research in the exercises throughout the day (stages 4 and 
5); and 97 per cent expressed the intention to use findings as they developed approaches and 
strategies for their setting (stage 6). 


Table I: Findings from the adapted ‘ladder of use’ scale (n=33) 


Stage 

Strongly 

disagree 

(%) 

Disagree 

(%) 

Neither 

(%) 

Agree 

(%) 

Strongly 

agree 

(%) 

Reception 

- 

- 

- 

58 

42 

Cognition 

- 

- 

6 

61 

33 

Discussion 

- 

- 

3 

70 

27 

Reference 

- 

3 

3 

49 

45 

Effort 

- 

3 

3 

63 

31 

Influence 

- 

3 

- 

45 

51 


To follow up on stage 6, we asked an open-ended question: ‘Why do you want to use the 
research?’ and inductively coded the responses. Here, the majority (46 per cent) suggested that 
use was related either to enhancing the ‘general’ or generic quality of their practice; meanwhile 
20 per cent wanted to use the findings to augment specific areas of their work that they felt 
needed improving. In addition, 6 per cent wanted to use the research to help them improve the 
practice of others in their setting. Other responses included a desire to better understand a 
particular aspect of child development (6 per cent) and to develop evidence-informed practice 
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(3 per cent), while one more cynical respondent suggested that the findings would be useful to 
back up changes that had already been made. 

Finally,we examined what else had been discussed within the group exercises. Here, responses 
centred mainly on ‘general discussion about the EYFS’ (32 per cent), with others subjects low 
in frequency, but ranging from ‘extending children’s interests’ to ‘notions of quality’. Importantly, 
however, just over a third of respondents (36 per cent) indicated that these discussions were 
pertinent and did influence how they developed ideas and strategies for their settings.The same 
number suggested that their strategies were also informed by their practitioner knowledge, that 
is knowledge of their children and setting. 


Adapted levels of use scale 

Findings from our combined expertise and levels of use scale appear in Table 2. In administering 
the survey we explained to participants that each question built on from the one before. If 
they felt they had reached a specific level already, participants were asked to agree (or strongly 
agree) with the statement. This process continued until respondents reached a level that did 
not reflect their engagement or behaviour in the way described. As can be seen, and as might 
be expected after four months of activity, the majority of respondents (94 per cent) indicated 
that they had made attempts to use at least some of the research presented as part of their 
day-to-day practice. Following this, however (and again, as might be expected given the likelihood 
that there will generally be fewer people at the highest levels of expertise), there is a gradual 
decline in the percentage of respondents agreeing or strongly agreeing with the statements 
regarding positive research use. For instance, as we progress through the questions: 94 per 
cent agree or strongly agree that their use was around ‘orientation’, that is in considering how 
research findings might be implemented; this falls to 77 per cent for ‘preparation’ (making plans 
to use findings); while 65 per cent suggest they are ‘proficient’ (beginning to tailor research 
findings so they are contextually appropriate).At the highest levels of expertise only 47 per cent 
agreed (nobody strongly agreed) that they were regularly adapting the strategies suggested by 
the research in order to make them even more effective (‘proficient+’); likewise only 35 per cent 
agreed (nobody strongly agreed) with the statement relating to full expertise; ‘I have used the 
research strategies we learnt about so often now that I barely need to even think about them; 
they’ve just become part of my day-to-day practice.’ 


Table 2: Findings by typology, derived from the combined expertise and Levels of Use scale (n=34) 


Descriptor 

Strongly 

disagree 

(%) 

Disagree 

(%) 

Neither 

(%) 

Agree 

(%) 

Strongly 

agree 

(%) 

Non-use 

35 

53 

6 

- 

6 

Orientation 

6 

- 

- 

88 

6 

Preparation 

- 

12 

12 

64 

12 

Novice 

- 

- 

29 

59 

12 

Competent 

- 

6 

19 

62 

13 

Proficient 

- 

- 

35 

53 

12 

Proficient+ 

- 

12 

41 

47 

- 

Expert 

6 

12 

47 

35 

- 
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Conclusions 

The findings presented here represent only an initial attempt to establish ways of measuring: (i) 
meaningful evidence use, defined as the result of combining formal and practitioner knowledge 
such that this leads to expertise; and (ii) the effectiveness of knowledge-creation activity as a 
means of establishing meaningful evidence-use. 

Nonetheless, based on the results of the two surveys outlined above, our findings would 
appear to be encouraging. Not only does knowledge-creation activity seem to provide an 
effective way of communicating research and keeping it ‘top of mind’, but it seems to enable 
practitioners to develop expertise in using evidence by helping them combine formal knowledge 
with practice-based or tacit knowledge in practical ways. Simultaneously, our interview data 
appears to triangulate and verify the trustworthiness of measures used in the surveys as a way 
of measuring levels of expertise in evidence use (Lincoln and Guba, 1985). It provides us with 
confidence that not only is our approach to measurement‘valid’ but, as a result, it meaningfully 
represents the level of expertise captured: we are confident in our claim that knowledge creation 
works to establish expertise in evidence use. In addition, by using both the ladder of use scale and 
an amended levels of use scale, we believe we have tackled the issues regarding ‘self report’ raised 
by Levin (2013) above. Namely, rather than examining participants’ perceptions of instrumental 
research use (what people believe they did) we ask participants to make statements in relation 
to their present behaviour - how they are currently responding in relation to an ongoing activity. 
This ensures we are looking at research use as process rather than event (Landry et al., 2003). 

At the same time, the approach we have developed to assess expertise in evidence use 
does have its limitations, namely that it is only really applicable to situations where researchers 
are working regularly with practitioners on areas of practice development, where the general 
desire is that these areas should become evidence-informed. In other words, our approach 
measures the effectiveness of activity to engage practitioners with evidence rather than, more 
generally, whether practitioners are expert evidence users. However, this does not mean that 
our approach has relevance only to relatively few instances of initiatives or activity. On the 
contrary, in school systems such as that in England - where the expectation is that schools or 
alliances of schools should lead their professional development activity often in partnership with, 
or drawing on the help of, universities - it is likely that these instances will soon be increasing in 
number (Greany, 2014). Similarly, the scale could be equally meaningful as a way to measure the 
extent to which practitioners have engaged with the findings of practitioner enquiry or action 
research.As such, we suggest that the scale we have set above will soon play a vital part in helping 
establish whether ‘self-improving’ school systems (Hargreaves, 2010) are also those that are 
meaningfully evidence-informed. 
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